Gradient Descent

Gradient Descent is a first-order iterative Optimization algorithm used to minimize a Cost Function by finding the local minimum of a differentiable function. In the field of Machine Learning, it is the primary method for training models like Linear Regression and Neural Networks. The algorithm relies on the Gradient, which is a vector of Partial Derivatives indicating the direction of steepest ascent. By taking steps in the opposite direction, the algorithm iteratively approaches the optimal Weights and Biases.

The size of these steps is determined by a Hyperparameter called the Learning Rate. If the Learning Rate is too large, the algorithm may overshoot the minimum; if it is too small, convergence will be inefficiently slow. As noted by IBM, the choice of learning rate is critical to the performance of the Optimization process. Different variations of the algorithm are employed based on data size, including Batch Gradient Descent, Stochastic Gradient Descent (SGD), and Mini-batch Gradient Descent. Further details on the mathematical convergence properties can be found on Wikipedia.

Advanced versions of Gradient Descent include adaptive methods such as Adam Optimizer, RMSprop, and Momentum. These techniques help navigate complex surfaces in Deep Learning where gradients might become vanishingly small or oscillate significantly. Practical implementation often requires Feature Scaling to ensure the Cost Function has a more spherical shape, facilitating faster convergence.

• Backpropagation
• Convex Optimization
• Loss Function
• Supervised Learning